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@ Graphical user interface management system and method. 

@ A graphical user interface management system comprises means for storing one ore more tables of a 
relational type each describing, as one or more objects, one or more graphical components displayed on a 
display apparatus, one or more procedure modules or procedure module groups, one or more relations among 
the graphical components and the procedure modules groups, or one or more relations among the procedure 
modules or procedure module groups. The system also comprises means for translating messages to the 
objects into formats of the con^sponding tables, performing predetenmined queries on tfie tables, and 
performing invocations of procedure modules or procedure module groups determined by the queries. 
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The present invention relates to graphical user interface management systems, and in particular lo 
those systems which enable management of user interfaces by means of tables of a relational type 

The practise of object oriented programming is becoming increasingly popular. Programming in an 
object onented language ,s generally carried out using a data abstract approach. The data abstract 
approach .s a programming technique in which data expressions and operations on data are combined and 
accesses to the data are allowed only from given interfaces of the operations. Data in such a program is 
expressed with common data expressions and interfaces. Generally in an object oriented language 
franieworks for expressing common data are referred to as " classes". Objects are entities of data having 
configurations such as classes. Application programs are written by definition of a plurality of classes 
generation of objects, and operations to the objects. Another feature of programming in an object oriented 
language is that classes have a hierarchy to cause inheritance between upper and lower classes 
Ttie above features allow higher level data modelling, and sharing and reuse of data types 
TTie use or definition of classes however requires the use of an object oriented language, and sharina 
and reuse of data types directed to a specific object oriented language can not be used with other object 
onented languages. Accordingly class definitions as well as existing files prepared in one specific object 
onented language can not be used in other object oriented languages. 

Many prototype user interface management system have been presented. For example " Coral" ( A 
o'^f f ?T'^ ^""^ Constraints. OOPSLA '88 Conference Proceedings 

pp.37-45) by P. A. Szekely and B. N. Myers is a user interface management system for producing graphiral 
objects on windows. Coral is written in an object oriented language (CLOS) which is based o7liSP 
language. Coral enables users to: 

1. Define graphical ejects using a declaration language. 

2. Set constraints among objects using a procedure language. 

The generation of new graphical object by users basically requires an understanding of CLOS which 
25 gives the essence to Coral, and the declaration language which Coral defines. Also, the description by users 
of constraints among the objects requires an understanding of CLOS and the procedure language 
For building user interface management systems. Coral enables users to: 

1. Prepare special variables for isolation between a portion for display of graphical objects and an 
execution portion which is invoked by the display portion. 
30 2. Prepare Class sets for dealing with a plurality of graphical objects in a set 

3. Define Procedures within the objects for identifying objects which are indicated by input actions and 
modificatons to data in the objects. 

TTie reason why the specific variable and class sets are needed as is that object oriented languages 
lack a concept of relation". Consideration of specific variable and classes must be introduced during 
35 design. =* 

It is also reasonable and complies with ideas of database systems that the object access functions of 
the feature 3 should be provided outside of the objects, because the objects are considered as data holding 
means. ^ 

Another prior art reference connected to the present invention is Japanese Published Unexamined 
Patent Application No. H1-229321. which discloses user interface generation tools which hoM location 
informafton of objects in tables. That prior art does not however suggest the use of tables of a relational 
nature to enable queries of a plurality of tables simultaneously. Neither does it disclose generation or 
management of graphical objects or procedure objects directly with tables. 

An object of the present invention is accordingly to provide a system for easy generation and 
management of objects of graphical user interfaces without use of any object oriented language 

In accordance with the present invention, there is now provided a graphical user interface management 
system comprising: storage means for storing one or more tables of a relational type each describing , as 
one or more objects, one or more graphical components displayed on a display apparatus, one or more 
procedure modules or procedure module groups, one or more relations among the graphical components 
one or more relations between the graphical components and the procedure modules or procedure module 
groups, or one or more relations among the procedure modules or procedure module groups; and. control 
means for translating messages to the objects into fomiats of the con-esponding tables, performing 
predetermined queries on the tables, and performing Invocations of procedure modules or procedure 
module groups determined by the queries. 

Viewing the present invention from a second aspect there is now provided, an object management 
system for graphical user interfaces comprising: basic object storage means for storing one or more tables 
of a relational type each describing, as one or more basic objects, one or more graphical components 
displayed on a display apparatus, or one or more procedure modules or procedure module groups- 
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complicated object storage means for storing one or more tables of a relational type each describing 
comphcated objects relating one or more of the basic objects to one or more oltiers of the basic objects- 
means tor stonng one or more tables of a relational types each describing, as one or more objects, one or 
more internal states of one or more application programs; and. control means for translating messages to 

5 the objects into formats of corresponding tables of the relational type, performing predetermined queries on 
the tables of the relational type, and perfonmlng invocations of procedure modules or procedure module 
groups determined by the queries. 

A preferred embodiment of the present invention will now be described by way of example only, with 
reference to the accompanying drawings in which:- 

D Figure 1 is a block diagram generally illustrating a user interface management system according to the 
present invention. 

Figure 2 is a block diagram illustrating a programming environment system for implementing the user 
interface management system of Figure 1 . 

Hgure 3 is a flow chart illustrating operations of the programming environment system of Figure 2 in a 
5 programming mode. 

Figure 4 is a flow chart illustrating operations of the programming environment system of Figure 2 in an 
execution mode. 

Figure 5 is a block diagram illustrating the configuration of the customer programming facility (CPF) 
shown in Figure 2. 

f Figure 6 is a block diagram illustrating the details of the event recorder shown in Rgure 5. 
Figure 7 is a drawing illustrating events to be dealt with by the event recorder. 
Figure 8 is a drawing illustrating the operations of the story editor shown in Figure 5. 
Figure 9 is a drawing illustrating the configuration of the user interface enabler (UIE) shown in Rgure 2. 
Figures 10, 11, and 12 are drawings illustrating the operations of the user interface enabler shown in 
Figure 2. 

Rgures 13 to 18 are drawings exemplifying the tables of a relational type used for the above 
embodiment. 

Figure s 19 to 21 are drawing s illustrating a modified version of the above embodiment. 

In the embodiment shown in figure 1, the present invention is applied to the management of objects 
used for a user interface of a programming environment system permits customisation of one or more 
applications and to produce a new application. 

Rgure 1 shows the object management architecture, comprising a base table group 10, a relationship 
table group 11. a transition table group 12, and a control program 13. The base table group 10 has one or 
more base tables each describing one or more basic objects. The basic objects are for general purposes 
and do not depend on semantics of applications. Tables consisting of attributes of sizes, colours and so on 
for graphical objects such as rectangles and annws are examples of base tables ( Box table and Arrow 
table as shown In Rgure 14). Tables defining procedures in connection with graphical objects are other 
examples of base tables ( Procedure definition table as shown in Rgure 13). Class variables and methods ( 
or operations) of object oriented languages can both be defined as attributes of tables in this architecture. 
There are two kinds of operations, that is. execution modules for calculation and for displaying objects, and 
operation modules for query of objects. 

The relationship table group 11 is comprised of one or more relationship tables, which tables are used 
for defining complicated objects by use of objects defined by the base tables. The relationship tables are 
defined so as to comply with application semantics. Screens each having a plurality of graphical objects 
are, for example, defined by relationship tables (Panel definition table as shown in Rgure 15). For enabling 
navigation witti a graphical object, another relationship table is used ( Refer to query lines in Expressions 1. 
2, and 3 described later). Since objects which do not depend on applications, and objects which depend on 
applications, are respectively defined by base tables and relationship tables, and are divided from each 
other, user interfaces can be easily designed and modified. 

The transition table group 12 comprises of one or more transition tables. The transition tables define 
transitions of internal states and panels of applications ( Refer to Panel transition table in Figure 21). A user 
action causes a transition from one state to the next, or one panel to the next, according to one of those 
tables. 

The control program 13 is designed for providing an interface between the aforementioned tables of a 
relational type and the outside. For example, the control program 13 convert a window message in the 
system queue into^ a table format, and accordingly keeps a corresponding conversion table ( Window 
message table in higure 18). The control program 13 is also provided with a query function with tables of 
the relational type ( Query lines in Expressions 1 ,2, and 3. later). 
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100?rpfererd%'he'LtrS:^^^^ °' ^ °" ^'^'^'^ programming env.Vonn.en, system 

Ml P im ! P^°9famm,ng environnnent system 100 comprises an application logic enabler 
(ALE) 101 a customer programming facility (CPF) 102. and a user interface enabler(U B IOS The 

J (IBM PS/55 are trademarks of International Business IVIachines Corporation US A) and is 
.n a system prograr. 105 consisting of an operating system and a pLentin ™Tr for 
managmg windows (hereinafter referred to as the presentation manager/operating system) ^ 
nr^^L '"T" ^"^""S application functions, and one or one set of applications on the 

propose office applications such as a spread sheet, a word processor, a data base manag? a LSSr 

Se^tr^^^^^ 

- «rrmodur(fjgi™ 

withThe suppo?<S°wS"u.Pr""'' "'V' P^^a^^-^-g environment system of the embodiment 
wim the support of which users can customize the applications of the ALE 101 to constitute a nnw 
application, and to associate the new application with a new interface layout 

exrh'^rn^J w ""T *° "'^ "on-existent. the CPF actually controls 

TaT^'ioI manager/operating system 105 and the appLTons 2 

ronr^irTT'"V"^'°"'"^' "^"'^"^ ''^^ ^ P'°9^^ '"O'^e customization and an execution 

:s::^^ZTj™~^^^^^ ---^ — 

«nniS«I! ^ ^r*"^ °* ^ P™^'^'" "^^ P^ogra-" "lode defines tasks to be executed on 

applicatons before the exeoutfon of the tasks in the execution mode. As shown in Rgui 3 in S mode 
the user acUjal y perfomis an operation on applications, which is then reconJed (81 1) Such ™ Jon fs 
^led recort A series of operations is saved as data called an event file. One or Lre sa^d evSTlel 
rin,' ? P^''.'^"?^^*^ ^"^ ot stories by a story editor 107 (in Rgure 5) c J^e of deSlq 

H ! K ■ '^^^'"^ to '^'■^e'^t tesks to be executed (813) The user 

Each step will be described later with reference to the drawings from Figure 5 onward 
Figure 4 is flowchart of an execution mode. The execution mode is a mode to reexecute the recorded 
ope ations using the user interface customized in the program mode. As shown in Figure 4 in hfe mSe 2 

^:^s siyssr °" '"'^ ''''' « ^° ^^^^i 

embSLeT"' """" "'""""'^ programming environment system of the 

Hgure 5 shows the configuration of the CPF 102 In Fiaure 5 th^ rPF in9 r^r.^.:*^^ * 
recorder .06. a story editor 107. and a linker 108. The "eveSt" Lord^ iT/recorraH: Im T 
= InTtrir^ i" ^ - --ded and reproduce them"* 


in the 


execu«on mode. The editor 107 suppo^s the us^ri; gere^ti^g'ITt of Terror 7^^^^ by 
combining those recorded event files. The linker 108 associates the event files o? sto^'i a gShIca' 
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object on the user interface newly generated. 

Figure 6 shows details of the event recorder 106. In Figure 6, the event recorder 106 consists of an 
event recording section 109, an event file storage 110, and an event play back section 111. 

The event recording section 109 has a function to monitor and control application program interfaces 
5 which an application program 112 uses to monitor user's behaviour. In this example, the event recording 
section monitors and controls information from the event queue 105b which is provided by the presentation 
manager 105a. That is, each time the application program 112 dequeues an event from the event queue 
105b, the event recording section intercepts the event, and stores the sequence of events in the event file 
storage 110 as an event file. Specifically, those events are intercepted by a function, called "input hook", 
70 provided by the presentation manager 105a. For details on the input hook, refer to "IBM Operating 
System/2 Programmer's Toolkit Version 1.1 Programming Guide", pp. 11-16 and pp. 11-18; (Operating 
System/2 is a trademark of IBM Corp.). In Figure 6. the reference number 105c corresponds to the 
operating system. 

Figure 7 shows the structure (A) of an event to be recorded and examples (B) wherein direct events 

15 provided by for example a mouse or a keyboard, and events produced by the system such as the initiation 
of a menu, a window or an application are manipulated. Such event information consists of a window ID 
(Window Handle) 113a, an event type (Message) 113b. and a time stamp (Timestamp) 113c, and other 
parameters (Parameters...) 113d dependent in meaning on each message. 

Referring now to Figure 6, there is an event play back section 111, which reproduces those events 

20 stored in the event storage 110 in the play back mode. When the application program 112 is about to 
dequeue an event from the event queue 105b, the event play back section 111 transfers a corresponding 
event in the event file storage 110 to the application program 112 as if the event came from the event 
queue 105b. Specifically, the event play back section 111 reproduces such events by use of a send 
function for sending a message (WinSendMsg) that is one of functions provided to achieve an inter- 

25 application communication function and the like. For details on the function of WinSendMsg, refer to IBM 
Operating System/2 Programmer's Toolkit Version 1 .1 Programming Guide, pp. 3-6 and pp. 3-8 (Operating 
System/2 is a trademark of IBM Corp.). 

Figure 8 shows an example of a user interface screen of the story editor 107. /Vs Figure 8 shows, the 
story editor 107 provides fields of an event list 107a, a story board 107b, and commands 107c, and enables 

30 events to be edited visually. The story editor 107 permits handling more than one event file in a lump. The 
registered event files can be referred to on the event list 107a. The event files in the event list 107a are 
copied on the story board 107b with the commands 107c. One of the most important functions of the story 
editor 107 is to combine more than one event file into a larger executable record. Moreover, in a story the 
commands 107c, the command field may be used to describe a control structure. In the following 

35 description, stories as well as event files are referred to as procedure modules. Stories are also held in the 
event file storage 1 10 in form of a relational type table as shown in Figure 13. 

The linker 108 shown in figure 5 is designed to connect the graphical objects of the user interface and 
procedure modules so as to operate procedure modules in response to the corresponding operators (such 
as a mouse click) to the user interface. The connection between the graphical objects and the procedure 

40 modules are established by the user, as described later, and the resultant connection is stored in tables of a 
relational type, that is, the object operation table in Hgure 16 and the procedure invocation table in Figure 
1 7. and query lines of Expressions 1 , 2, and 3 which are described later. Those tables of a relational type 
and query tines are prepared and kept using UIE 103, and the detailed description will be given together 
with UIE 103 below. 

45 The UIE 103 is a graphics editor for defining a user interface. The display section is similar to a 
graphics editor of the well-known WYSIWYG 0/Vhat you see is what you get) type, so that users are allowed 
to use graphical components as they like in displaying a user interface. The functional part of this graphics 
editor is the layout editor 114, by use of which a user may define the appearance of a new user interi'ace at 
will for use in the future. 

50 Figure 10 is an example of a layout generated by the layout editor 114. In this example, 1 16 to 120 are 
graphical objects constituted by grouping character strings and rectangles. Each graphical object is called a 
layout object, which is associated with a procedure module. Figure 10 shows an example of a process of 
office work from writing to printing of a report created by the UIE 103. The arrow 121 here is not associated 
with any procedure module, but only serves as a guide for operational procedures. (Yet, the arrow 121 may 

55 be associated with some procedure module, if necessary.) 

These definitions of graphical information are handled in the form of layout file 115 (Figure 9) that is 
used for saving the user interfaces. The layout file 115 comprises, in particular, the box and arrow tables ( 
base tables) in Figure 14 and the panel definition table ( relation ship table) in Rgure 15. 
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The following describes how to associate these layout objects with procedure modules. Figure 11 is an 
example of linking, and Rgure 12 shows its procedure. Referring now to Figures 11 and 12. a particular 
graphical object 122 to be linked is first clicked {S31), whereby the graphical object is related to a required 
transition as shown in Figure 16. On clicking one graphical object 122, one new tupple is added to the table 
of Figure 16. The transitions are of either internal states or panels of the application. The clicked object is 
highlighted so as to ensure the use to be informed (S32). In this connection, Figure 16 illustrates v/fiich 
transition is caused upon providing a particular operator to a particular graphical object, tn this example . a 
mouse click operation is a default parameter for operators. 

Next, when the "LINK" action is selected on the menu 123 of the UIE 103 (S33), another menu 124 will 
appear to select "STORY" or "EVENT" (S34). In this example, the STORY is selected (S35). Then, a story 
list dialog box 125 appears (536). By selecting required procedure module from the list with click (337). a 
link is established (S38). 

Unking may be attained In a similar operation in case where a link from the layout object is repasted to 
another procedure module. 

Furthermore, it is possible automatically to execute a procedure module associated with one layout 
object after executing another procedure module associated with another layout object by associating those 
layout objects with each other. 

A user interface generated in this way has an appearance like that in Figure 10. invokes a procedure 
module in response to a user's operation, supplies events sequentially to applications, and automatically 
executes the user's desired tasks. For example, when a mouse click action Is provided on the object 
obj002, the control program operates as follows: 

1. A window message is converted into an entity of the window message table in Figure 18. 

2. The graphical object which Is subjected to the mouse click action is searched, and the entity having 
"obj002" as the object ID is selected. 

The function of the control program for this phase is described by the following pseudo code. 

(Expression 1) 


SELECT Object ID FROM Box Table OR Arrow Table 
WHERE Parameter 1 of Window Message Table is inside 

the rectangle identified by the location of Panel 
Definition Table and the size of Box Table 

3. Since the combination of the selected graphical object and the input may cause a transition of the 
application, the object operation table is checked for ''obj002". In this case, a mouse click action for 
"obj002" is determined as to raise the transition Tr002. 

The function of the control program for this phase is described by the following pseudo code. 


(Expression 2) 


SELECT Transition ID FROM Object Operation Table 


IVHERE the operator of Object Operation Table = Message ID of 
Window Message Table AND Object ID of Object Operation 
Table = Object ID selected in Expression (1) 


4. The procedure table is consulted for determining what the selected transition is to do. Tr002 initiates 


the procedure Pro002. 

The function of the control program for this phase is described by the following pseudo code. 


5 (Expression 3) 

SELECT Invocation FROM Procedure Invocation Table 
WHERE Transition ID of Procedure Invocation Table = 
Transition ID selected in Expression (2) 

CALL Invocation just selected 

IS 

As understood from Figures 16 and 17, in the above example, the graphical objects to be clicked 
directly correspond to the procedure modules to be invoked, and there is accordingly little need to describe 
and manage transitions among internal states of the user Interface with regard to invocation of procedure 

20 modules additionally. It is however critical to define the transitions and design a dynamic scenario of the 
user interface in connection to generation or management of objects for a user interface. Besides, a user 
interface as shown in Figure 19 requires to describe and manage the internal states of the user interface 
additionally, because while transitions correspond to procedure modules to be invoked one to one.graphicai 
objects to be clicked don't necessarily correspond to procedure modules to be invoked one to one. in the 

2s example as shown in figure 19, a click action of the print box in the procedure panel causes a print panel 
window to open and enables to input print parameters, and a click action of enter button on the print panel 
causes invocation of a print procedure. For the example of Figure 19. a panel table and a procedure 
definition table illustrated In Figure 20 are prepared as well as a panel transition table and a procedure 
invocation table illustrated in Rgure 21 . 

30 Advantageously, the present invention permits objects of the user interfaces to be more easily 
generated and managed tiirough the use of tables of a relational type. 

Ctaims 

36 1. A graphical user interface management system comprising: 

storage means for storing one or more tables of a relational type each describing , as one or more 
objects, one or more graphical components displayed on a display apparatus, one or more procedure 
modules or procedure module groups, one or more relations among the graphical components, one or 
40 more relations between the graphical components and the procedure modules or procedure module 
groups, or one or more relations among the procedure modules or procedure module groups; and, 

control means for translating messages to the objects into formats of the conesponding tables, 
performing predetermined queries on the tables, and performing invocations of procedure modules or 
45 procedure module groups determined by the queries. 

2, A system as claimed in Claim 1, wherein the graphical components include one or more panel layouts, 
and boxes, arrows, and/or other graphical components which constitute the pane! layouts. 

so 3. A system as claimed in Claim 1 , wherein the procedure modules include event streams input to one or 
more application programs. 

4. A system as claimed in Claim 1, wherein the procedure modules Include function calls for input to one 
or more application programs. 

55 

5. A system as claimed in Claim 1 , wherein the procedure modules include query routines. 

6. A graphical user interface management system as claimed in claim 1. 
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7 . A system as ciaimed in claim 6 wherein tho storage means comprises: 

a basic object storage portion for sorting one or more tables of a relational type each describing, as 
one or more basic objects, one or more graphical components displayed on a display apparatus, or one 
or more procedural modules or procedure modules groups: 

a complicated object storage portion for storing one or more tables of a relational type each 
describing one or more complicated objects relating on or more of the basic objects to one or more 
others of the basic objects; 

8. A system as claimed in claim 6 wherein the storage means comprises:- 

a first object storage portion for storing one or more tables of a relational type each describing one 
or more objects of a graphical user Interface which objects are independent of one or more application 
programs; 

a second object storage portion for storing one or more tables of a relational type each describing 
one or more objects of the graphical user interface which objects are dependent on the application 
programs; 

9. A graphical user interface management method comprising steps of: 

storing one or more tables of a relational type each describing, as one or more objects, one or 
more graphical components displayed on a display apparatus, one or more procedure modules or 
procedure module groups, one or more relations among said graphical components, one or more 
relations between said one or more graphical components and said one or more procedure modules or 
procedure module groups, or one or more relations among said procedure modules groups; 

translating messages to said objects into formats of corresponding tables of said relational type; 

performing predetermined queries on said tables of said relational type; and. 

performing invocations of procedure modules or procedure modules groups determined by said 
queries. 

10. A graphical user interface management method of Claim 9. wherein said graphical components include 
one or more panel layouts, and boxes, an^ows, and/or other graphical components which constitute said 
panel layouts. 

11. A method as claimed in Claim 9, wherein said procedure modules include event streams input to one 
or more application programs. 

12. A metiiod as ciaimed in Claim 9, wherein said procedure modules include function calls inputted to one 
or more application programs. 

13. A method as claimed in Claim 9. wherein said procedure modules query routines. 

14. A method as claimed in Claim 9 comprising step of: 

storing one or more tables of a relational types each describing, as one ore more objects, one or 
more internal states of one or more application programs; and, 

15. A method as claimed in claim 14 the steps of: 

storing one or more tables of a relational type each describing, as one ore more basic objects, one 
or more graphical components displayed on a display apparatus, or one or more procedure modules or 
procedure module groups; 
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storing one or more tables of a relational type each describi:i£ cifc or more <;oinplicated objects 
relating one or more others of said basic objects; 

16. A graphical user interface management method as claimed in claim -.4 comprising steps of: 

storing one ore more tables of a relational type each describing one or more objects of a graphical 
user interface which objects are independent of one or more application programs; 

storing one or more tables of relational type each describing one or more objects of said graphical 
10 user interface which objects are dependent on said application programs; 
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